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APPARATUS FOR GENERATING CONTENT CODES 
FOR AUDIOVISUAL PROGRAMS BY MULTIPLE VIEWERS 

5 

RELATED APPLICATIONS: 

The subject matter of this application is related to commonly owned 

application S.N. , filed , and entitled APPARATUS 

FOR GENERATING CONTENT CODES FOR AUDIOVISUAL PROGRAMS, 
10 incorporated herein by reference. 

BACKGROUND OF THE INVENTION 

A. Field of Invention 

15 This invention relates to the tagging of audiovisual content of a 

program, such as analog or digital motion pictures that are transmitted over 
cable, or video transmitted over the Internet. More particularly, it pertains to 
an apparatus for tagging individual scenes of the audiovisual program by 
several people so that it can later be replayed with selected scenes being 

20 omitted or replaced by others. 

B. Description of the Prior Art 

Presently, many video programs have ratings that can be used by 
viewers to determine whether they would like to play the programs, or whether 
25 they would allow others (such as children or adolescents) to play them. 

These types of ratings are also becoming available for TV shows, games, and 
other audiovisual programs. Programs broadcast over the Internet may be 
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filtered or otherwise restricted using on-line filtering services, and set-top 
boxes and TVs with V-CHIP content filtering technology also exist for this 
purpose. 

As an improvement to current technology, it has been suggested that 

5 rating codes be provided for individual segments, or scenes, of programs. 
Automated playing devices would then be able to seamlessly eliminate 
segments with objectionable content, and/or substitute other segments with 
acceptable content if such segments are available. Commonly assigned co- 
pending application S.N. 09/479,819 filed January 7, 2000, entitled "Content 

10 Control of Broadcast Programs" and incorporated herein by reference, 
discloses a personal video recorder that receives and stores a broadcast 
program with each scene being tagged with a content rating. A viewer 
designates parameters for acceptability of content, and the appropriate 
scenes are played as a seamless continuous program. U.S. Patent Nos. 

15 5,987,21 1 and 6,208,805 to Abecassis discloses an apparatus that records a 
program on a disk and filters content according to viewer preferences. A 
viewer is shown only the segments having acceptable content. A parent has 
the ability to review a program and tag unacceptable scenes. The apparatus 
then omits the unacceptable scenes when the program is played. 

20 A major disadvantage of all of these systems is that, whether a 

universal rating is assigned to a whole program, or ratings are assigned to 
each of a program's individual scenes, the ratings still reflect a limited number 
of categories accepted by the public at large, industry-wide guidelines or the 
opinions of an editor. Thus, they fail to reflect inherent differences in the 
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filtering categories of individual customers due to many factors such as 
geographic location, level of education, income, religious and moral beliefs, 
etc., factors that meaningfully affect a person's views and preferences for 
what kind of programs and content they, or their children, should view. 
5 A further disadvantage of prior art systems is that typically a single 

person provides the content codes. Therefore, the content codes selected by 
that one person reflects his/her scene characterizations that may or may not 
match the characterizations of others. What may be a "violent" scene to one 
person may not be such a scene to another. 

10 

OBJECTIVES AND SUMMARY OF THE INVENTION 

It is believed to be beneficial to allow a group of viewers to tag 

programs based on composite characterizations of scenes. 

Accordingly, it is an objective of the present invention to provide an 
15 apparatus that permits viewing of scenes of a program, and the tagging of 

scenes by a group of viewers with content codes that can be used later for 

discriminating purposes. 

A further objective is to provide an apparatus that is user-friendly and 

easy to operate. 

20 A further objective is to provide an apparatus in which a program's 

scenes can be designated by the program provider, automatically detected, or 
designated by a viewer. 

Briefly, viewers display a program on an audiovisual monitoring station 
system. The apparatus for assigning content codes to the program's scenes 
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constructed in accordance with this invention includes a scene detector 
identifying the individual scenes of a program. The scene detector, in one 
embodiment, detects scene changes by analyzing the characteristics of 
sequential frames using known techniques. In another embodiment, the 

5 program itself includes codes that identify its scenes. In a third embodiment, 
the scene detector responds to a scene selector that is viewer activated. 

Also included in this invention is a scene indicator, which may be in the 
form of a visual or audible signal, and which indicates the beginning of a new 
scene. Viewers in a group are prompted by the scene indicator to enter 

10 respective individual content codes for the new scene. This content code is 
determined by the viewers based on the visual as well as the audio content of 
the respective scene. Once the individual content codes are entered by all the 
viewers, the scene indicator is deactivated until the detection of the next 
scene. Alternately, the scene indicator may be activated only for a brief 

15 period of time. A timer may also be provided to indicate the time left until the 
end of the scene (if known). If a viewer enters no content code by the end of 
the scene, a default value is automatically assigned for that scene for that 
viewer. The default value may be the individual content code for the 
immediately preceding scene. 

20 A code selector is also included in the apparatus for each viewer. This 

code selector is adapted to receive the individual content codes for a scene. 
An algorithm is used to determine a final content code based on the individual 
content codes. For example, the content code representing the most 
objectionable content may be selected as the final content code. 
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Alternatively, a straight arithmetic or weighted averaging method may be used 
to determine the final content code (where the codes are ranked in a 
predetermined order). This voting approach allows a committee of parents, 
for example, to code a program for a class of students in the least 

5 objectionable way, based on both the visual and audio contents of each 
scene. Of course content codes may be designated based on a variety of 
criteria independently of whether a scene may be objectionable or not. 

For each scene, a two-component code (two record fields) is 
generated which includes the final content code and an identification of the 

10 scene. The scene identification may be generated by a timer that times the 
duration of each scene as well as previous scenes and then generates a 
starting and ending time for the scene. 

The apparatus also includes a memory for storing the codes for all the 
scenes including the assigned content codes and scene identifications. 

15 The apparatus further includes a controller adapted to selectively 

activate the scene indicator and generally manage the generation and storage 
of the scene codes. 

In this manner several viewers can view a program simultaneously and 
select the content codes for each scene, as described. Alternatively, the 

20 viewers can view the program separately and independently of each other. 
The individual content codes are logged and the final content codes are 
determined any time after the last viewer has made his selections. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 shows a block diagram of an apparatus for assigning content 
codes to scenes by a single viewer; 

Fig. 2 shows a block diagram of the apparatus of Fig.1 modified to 
5 accept code selections from several viewers simultaneously; 

Fig. 3 shows a somewhat diagrammatic view of a selector panel on 
which a viewer can enter his preferences; 

Fig. 4 shows a flow chart for the operation of the apparatus of Fig. 2; 

and 

10 Fig. 5 shows a flow chart for generating content codes by several 

viewers using the apparatus of Fig.4. 



DETAILED DESCRIPTION OF THE INVENTION 

An apparatus for generating a content code by a single viewer is 

15 described in the above-identified co-pending patent application S.N. 

. In order to provide a better understanding of the present invention, 

this apparatus is first described herein in conjunction with Figs. 1-3. The 
apparatus 10, as shown in Fig. 1 includes a device such as a program player 
12 that is adapted to play an audiovisual program on an audiovisual 

20 monitoring system 14. The program could be recorded on a DVD disk, a VCR 
tape or other similar media, or in a personal video recorder receiving the 
program from a program source 16, e.g. through a cable, XDSL, or satellite 
connector. 

Associated with the player 12 are a scene detector 18 and a program 
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timer 20. The scene detector 18 is used to detect scenes in the program 
signal generated by player 12. The program timer 20 generates a count that 
can be used to provide unique identifications for the scenes of the program. 
Instead of the timer, other scene identification means may be used, such as a 

5 frame counter. 

A controller 22 directs the operation of the apparatus 10. The controller 
22 is coupled to the scene detector 18 and timer 20, and it is also associated 
with a code selector 24, a scene indicator 26, scene timer 27 and a memory 
28. It should be understood that all the elements in Fig. 1 can be 

10 implemented as a microprocessor-based device such as a PC, or can be 
incorporated into the program player 12 (especially if the player is a personal 
video recorder). However the apparatus 10 is shown in Fig. 1 as having 
discrete components for the sake of clarity. 

In the present invention, the apparatus of Fig. 1 is modified to allow 

15 several viewers to provide content codes for each scene of a program. The 
modified apparatus 10A is shown in Fig. 2. As can be seen in this Figure, the 
apparatus 10A differs from the apparatus 10 in Fig. 1 in that it is provided with 
several code selectors, such 24A-24D. (Fig. 2 shows four selectors; however, 
any number of selectors may be used.). Therefore, with this apparatus 

20 several viewers decide the code that is to be assigned to each scene based 
on its audio and visual contents. Each viewer operates one of the selectors. 

Details of one of the code selectors 24A are shown in Fig. 3. The 
code selector 24A consists of a housing 30 that supports a plurality of 
switches 32, 34, 36 and 38. Each of the switches may be labeled with a 
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content code, for example G, PG, R or X. (More precise content codes may 
be used, giving rise to more than just four switches. For example, content 
codes may be provided which describe more accurately the contents of the 
scenes, in terms of the amount or degree of sex, violence, nudity, profanity, 

5 etc.) Each of the switches 32-38 may be a momentary pushbutton switch. 
The scene indicator 26 may be positioned on the housing 30 as well. 

The apparatus of Figs. 2 and 3 operates as indicated by the flow chart 
of Fig. 4. In step 100 the program player is activated to start playing a 
program on audiovisual monitoring system 14. The program timer 20 is 

10 activated as well. The monitoring system 14 is arranged to allow all the 
viewers to see and hear the program. 

As the program is shown to the viewers, the frames forming the 
program are monitored by scene detector 18 to detect the start of scenes that 
comprise the program. Scene detectors of this kind are well known in the art. 

15 When the first scene is detected (step 102), the controller 22 logs its starting 
time as indicated by program timer 20 (step 104). 

Next, the scene indicators 26 are activated on each code selector 24A- 
24D (step 106). Each scene indicator 26 may be a simple light that is turned 
on or a beeper that is activated. The scene indicators 26 are activated to 

20 prompt each viewer for a content code, using the respective code selector 
24A-24D. 

In some instances, the scene detector 18 may be too sensitive for the 
purposes of the present invention. More particularly, the scene detector may 
designate every camera angle change as a new scene. Therefore, it may be 



-8- 



advisable to have one of the viewers determine the start and end time of each 
scene. For this purpose, one of the code selectors, for example, the code 
selector 24A of Fig. 3, may include a manual scene selector 40. The manual 
scene selector 40 may be a momentary pushbutton like switches 32-38. 
5 When this manual scene selector 40 is provided, the scene detector 18 can 
be turned off or omitted. 

In such an embodiment, after the program starts, the viewer associated 
with code selector 24A selects the start time of a scene by activating the 
manual scene selector 40. When the manual scene selector 40 is activated, 

10 the scene indicators 26 for each code selector 24A-24D are also activated, 
thereby indicating that the current scene has not yet been assigned a content 
code by the respective viewers. The start time is also logged. 

As discussed above, as each scene is displayed on audiovisual 
monitoring system 14 with audio heard through associated speakers (not 

15 shown), the scene indicators 26 are activated. As the viewers view and listen 
to the current scene they each make a determination of what its content code 
should be. They then designate for the scene their selected content code by 
activating the appropriate switch 32-38. (It is possible to activate multiple 
switches if the system accommodates multiple content codes for a scene.) As 

20 each viewer selects his or her individual content code, the scene indicator on 
the respective code selector 24A-24D is turned off. (Alternatively, If the scene 
indicator 26 is a beeper, it may emit a different sound than when it was 
activated in step 106). In an alternative embodiment, the scene indictor 26 
may be turned on or activated only for a brief period (e.g., three - ten 
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seconds). In this embodiment, the scene indictor 26 is deactivated 
independently of the selection of a content code by the viewer. The individual 
content codes are stored in memory 28. 

Returning to Fig. 4, in step 108 the controller 22 checks if all the 

5 viewers have selected individual content codes. If content codes have been 
selected by all viewers, then in step 1 10 the final content code is determined 
from the individual content codes using a predetermined algorithm as 
described above. This final content code is then logged together with a 
corresponding scene identification, e.g., its starting time code. 

10 In step 112 the controller 22 checks the scene detector 18 for a new 

scene. When a new scene is detected, the controller 22 logs the end (e.g., 
another time code) of the last scene (step 114). 

Next, in step 116 scene indicia, including the appropriate final content 
code(s) and the corresponding scene identification data, are stored in scene 

is content code memory 28. 

In step 118 a test is performed to determine if the end of the program 
has been reached. If the program end has not been reached, then the start 
time of the new scene is logged in step 104 and the new scene is processed 
as discussed above. 

20 In step 1 18 if the program end is detected, then in step 120 all the 

scene indicia are recorded in a specific file that may be left in memory 28, 
stored on the medium which itself contains the program, or somewhere else. 

Back at step 108, if a content code has not been selected yet for the 
current scene by at least one of the viewers, then in step 122 the controller 22 
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checks the scene detector 1 8 for a new scene. If a new scene is not 
detected, then the system 22 goes back to step 108. If a new scene is 
detected in step 122, then in step 124 a default value is assigned for the 
individual content code of the respective viewer(s). For example, the default 
5 code may be the code assigned by the respective viewer to the previous 
scene. 

In step 126 the end of the scene being processed is logged and the 
process continues with step 1 1 6. 

The apparatus 10 can be modified for other types of operation as well. 
10 In one embodiment the scenes may be already defined by a specific code 
embedded within the program itself. In this case, instead of a scene detector, 
a decoder detects the beginning and end of each scene. This alternative is 
indicated in Fig. 1 for element 18. 

If the beginning and end points of a scene (and therefore its duration) 
15 are known, then, at the beginning of each scene, a scene timer 27 may also 
be activated. This scene timer 27 can be configured to count down and 
indicate to a viewer how much time is left to the end of the scene (i.e., the 
time left before the viewer has to select a code). 

Instead of using an automatic or even manually operated scene 
detector, it is possible simply to allow the viewers to enter content codes at 
any time. Any change in content codes would in effect signal a new scene. 
The advantage of identifying the start of a real scene, however, is that the 
entered content codes are retroactive to the start of the scene. On replay, an 
appropriate player could curtail viewing of an objectionable scene from its 
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start, not just from when the content codes were entered. 

This voting approach allows a committee of parents, for example, to 
code a program for a class of students in the least objectionable way. 

Instead of designating whether scenes contain objectionable materials, 
content codes can be selected based on various other criteria. For example, 
all the scenes of a documentary program may be selected if they show 
animals. 

In another embodiment of the invention, even the apparatus of Fig. 1 
with only one call selector can be used to obtain a group consensus. The 
viewers use the same apparatus individually, at different times. As shown in 
Fig. 5 for this mode of operation, each viewer selects an individual content 
code which is logged in the same manner as described above. Thus steps 
200-226 are similar to corresponding steps 100-126 of Fig. 4. However, in 
this embodiment after the program is completely played for one viewer and 
the scene indicia are recorded (step 220), a check is performed in step 228 to 
determine if the viewer who has just coded the program is the last viewer. 
This check may be accomplished in a number of different ways. For example, 
when the program is played the first time, the controller 22 can be provided 
with the number of viewers that will review the program. Alternatively, a 
special control signal may be provided to the controller 22 when the last 
viewer has coded the program. 

Next, in step 230 the final content code for each scene is determined 
based on the individual content codes. In step 232, the final content codes 
are then stored. 
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By generating a content code for each scene of a program, a table is 
created which associates each scene with its content code and the address of 
the next scene. This table may be stored separately or may be attached to 
the program itself. The table can be used to control the manner in which the 
program is played. For example, co-pending commonly assigned U.S. 
application S.N. 09/479,819 filed January 7, 2000, entitled CONTENT 
CONTROL OF BROADCAST PROGRAMS and incorporated herein by 
reference discloses a method and apparatus wherein a program is broadcast 
together with its table to a customer with an appropriately configured personal 
video recorder. The customer selects an acceptable content code. The 
personal video recorder then can record and play only the scenes that have a 
predetermined content code, while skipping the scenes with other codes. 
Since the table with the content codes is received in advance, the personal 
recorder can determine the scenes that are to be skipped, and the duration of 
these skipped scenes. The recorder therefore can delay playing the program 
by the duration of the skipped scenes so that the program can be played 
without interruption. 

Another advantage of providing a program with a table of scene- 
associated content codes is that it allows a content provider to create a 
program with several interchangeable scenes, each scene having a unique 
content code. A customer may then elect to get a program with all the 
scenes. Alternatively, a customer may elect to a version of the program with 
all the scenes having a predetermined content code or codes. 
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Obviously, numerous other modifications can be made to this invention 
without departing from its scope, as defined in the appended claims. 
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